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ABSTRACT 



This report describes a technique which can be used 
to provide both formative and summative evaluation of television 
programs designed to promote educational development in 3™, 4-, and 
5-year-old children. The daily half-hour program evaluated is part of 
an Appalachia Educational Laboratory project, now in its second year. 
(Other project components are weekly visits in homes by 
pare professionals, and weekly visits by a mobile classroom.) Of the 
170 TV programs transmitted in the first two years, observational 
data was collected on 133. The telecasts, designed to elicit overt 
responses from children, made possible a systematic observational 
evaluation of viewer responses. Each of eight paraprofessionals 
watched the program with a different child every morning so that, in 
a random fashion, all 270 children were observed an equal number of 
times. The observer coded children's responses while viewing the TV 
program and rated them on a standard tally sheet. First year and 
second year programs were compared on (1) the ratio of responses to 
questions asked by the TV teacher, (2) the ratio of negative 
reactions to enthusiastic reactions, and (3) the average number of 
enthusiastic reactions. Results indicate that systematic 
observational systems can provide formative evaluation of preschool 
TV programs to guide changes in presentation techniques, content, and 
emphasis. A summary of the AEL Early Childhood Program is available 
as PS 004 889. (AJ) 
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ANALYSIS OF CHILDREN'S REACTIONS TO AEL ' S 
PRESCHOOL TELEVISION PROGRAM* 

The Appalachia Educational Laboratory has recently completed its 
second year of a three-year project to develop a viable alternative to con- 
ventional kindergartens which are prohibitively expensive in mountainous 
terrain. This alternative consists of a daily half hour television program, 
Around the Ben d, weekly visits in homes by paraprof essionals , and v/eekly 
visits by a mobile classroom. Assuming that instruction is most effective 
when the learner is an active participant, the activities in all three com- 
ponents are designed to increase the involvement of the three-, four-, and 
five-year-olds participating in the project. 



METHOD 



This report describes a technique which can be used to provide both 
formative and summative evaluation of television programs designed to pro- 
mote educational development in three-, four-, and five-year-old children. 
As one might expect, the television component hats the greatest difficulty 
in promoting this interaction. Therefore, an effort was made to identify 
those activities and techniques which are most effective in eliciting overt 
responses from the viewers. The television program focuses around a young 
woman, "Patty," who, as a friend of the viewers, has experiences and does 
things in her home which viewers often do at the same time in their homes. 
The activities are designed to foster a strong personal relationship be- 
tween Patty and the viewers. This, and the tendency of preschool children 
to react overtly during the telecast, makes it possible to use a systematic 



*This report was prepared by Dr. George Miller, who is responsible 
for formative evaluation of the AEL Early Childhood Education Program and 
works out of the AEL field office at Beckley, West Virginia. 
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observational system to evaluate the effect of each program on the viewers. 

The categories of this system and the basic ground rules are presented in 
Table 9-1. 

The project employs eight paraprofessionals , each of whom watch the 
television program with a child every morning in his home. The program is 
observed with a different child each day. In a manner as inconspicuous as 
possible, each paraprofessional codes the behavior of the child she is ob- 
serving on the code sheet presented as Attachment 9-1. The columns indicate 
the categories of behavior. The 28-minute program is divided into five, 
five-minute intervals and one, three-minute interval as indicated by the rows. 

Everytime the television " teacher n makes a suggestion, asks a question, 
or attempts to elicit a response from the viewer, the home visitor makes a 
tally in one of the first three columns. This tally indicates whether the 
viewer responded verbally, non-verbally , or not at all. The remaining four 
columns represent viewer behaviors that were not elicited. A tally is made 
in the appropriate column each time one of these behaviors occur. To the 
right of the matrix are the numberals 0, 1/4, 1/2, 3/4, and the word ALL. 

At the end of each five-minute interval, the paraprofessional circles the 
figure that most closely represents the amount of time the viewer had his 
eyes on the television screen. 

At the bottom of the code sheet is a place to write remarks. This 
area is used to describe unusual circumstances occurring during the program 
such as prompting by the mother, a paddling, or anything that had a signifi- 
cant positive or negative effect on the viewer. Reactions, in behavioral 
terms, to specific segments are also written here. 

The group of children observed watching the program each week is ro- 
tated so that, in a random fashion, all 270 children in the sample are 
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TABLE 9-1 

TELEVISION VIEWER BEHAVIOR: AN OBSERVATIONAL SYSTEM 



1. Physically Responds to Suggestions, Directions , or Questions : The viewer 

dances, paints, or moves as suggested or directed by the television 
"teacher" or shakes his head yes or no to a question. 

2. Verbally Responds to Suggestions, Directions , or Questions : The viewer 

responds by saying something; repeating a poem, words, or letter? answers 
yes or no. 

3. No Response to Suggestions, Directions, or Questions : The viewer does 

not comply as requested by the television "teacher" either physically or 
verbally. 

4. Verbal Enthusiasm : The viewer says something that indicates he is excited 

about something in the program. This can be a sound of glee as well as 

an intelligible word. 

5. Non-verbal Enthusiasm : Physical motions such as the clapping of hands 

that indicate the viewer is excited about something in the program. 

6. Verbal Indication of a Negative Reaction : The viewer yawns aloud, says 

words or makes sounds that indicate disgust, boredom, or a negative 
feeling about the program. 

7. Non-verbal Indication of a Negative Reaction : The viewer looks away from 

the television screen, leaves the room, plays with a toy or engages in 
other actions indicating disgust, boredom, or a negative reaction to the 
program. 

GROUND RULES : 

1. Whenever verbal and non-verbal behavior occur simultaneously, the tally 
is placed in the verbal category. 

2. When the television "teacher" requests the viewer to engage in a sequence 
of behavior or say a sequence of words or letters, doing the whole sequence 
results in only one tally. 

3. When the observer is not sure the television "teacher" has requested an 
overt response, no tally is made in the first three categories. If, 
despite the vagueness of the "teacher's" remark the viewer responds, 
this behavior is coded in category 4 or category 5. 

4. The observer initially encourages the viewer to watch the program with 
her but does not coerce him. However, despite the actions of the viewer, 
the observer watches the whole program and gives the impression of being 
quite interested in it. 
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observed an equal number of times. The eight code sheets for each day are 
then compiled to form one matrix that is interpreted. The following informa- 
tion can be found for each five-minute intervals as well as for the program 
as a whole : 

1. The proportion of questions , suggestions, and directions 
that elici.ted a verbal or non-verbal response from the 
viewer . 

2. The proportion of unelicited positive behavior to un- 
elicited negative behavior of the viewer. 

3. The average number of acts of unelicited positive behavior. 

4. The proportion of time spent by the viewer with his eyes 
on the television screen. 

It must be remembered that the data are taken from observing only eight 
viewers of each television program. Thus, the data are unreliable in that 
conclusions cannot be drawn from one observation. However, when a particular 
programming technique which occupies most of a f ive-minute interval is used 
on several occasions and that interval of time consistently rates high or low 
on one of the calculated variables, a decision regarding its continued use 
can be objectively and reliably made. When used in this manner, the data 
gathered using this observational system can be used to answer formative 
evaluation questions such as those below: 

1. What types of questions and techniques of questioning 
are most effective in eliciting verbal responses? 

2. What is the optimum number of questions that should be 
asked in a given time interval, given a specific 
purpose? 

3. What activities are most effective in eliciting verbal 
and non-verbal indications of enthusiasm? 

4. What types of music and songs are most effective in 
getting viewers to dance and sing? 

5. What techniques are most likely to get viewers to re- 
peat the sounds of letters and learn given cognitive 
skills? 
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6. What camera techniques, types of animation, and method 
of monologue are best in eliciting the desired effect 
on the viewer? 

7. What are the optimum lengths of various activities in 
terms of achieving given objectives? 

8. What types of stories should be read and what types of 
presentations are most effective in maintaining interest? 

Finding answers to these types of questions is made easier by having 
written scripts of each program broken into five-minute segments or by hav- 
ing a secretary record a description of what is taking place during each 
segment. Recording the number of questions, suggestions, and directions 
stated by the television "teacher" enables the checking of coder reliability. 

SUMMATIVE RESULTS 

Of the 170 programs transmitted during the second year, observational 
data of viewer responses were collected on 133 programs. These included 37 
programs produced during the first year and 96 produced during the second 
year. These two groups of programs were compared using (1} the ratio of re- 
sponses to questions, (3) the ratio of negative reactions to enthusiastic 
reactions, and (3) the average number of enthusiastic reactions. 



TABLE 9-2 

COMPARISON OF FIRST AND SECOND YEAR PROGRAMS ON THREE VARIABLES 



Variable 


1st yr. x programs 


2nd yr. x programs 


t* 


sign 


Response Ratio 


59.22 


73.53 


4.86 


.001 


Enthus .Ratio 


59.65 


73.22 


3.46 


.001 


Ave # of 










Enthus . Res. 


5.46 


8.38 


3.44 


.001 



y t-test of difference between uncorrelated means. 
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Although only the best of the 170 programs produced during the first 
year were repeated, the programs produced during the second year were sig- 
nificantly more effective in eliciting responses, maintaining a positive 
attitude toward the program, and generating enthusiasm. The data also in- 
dicated age and sex differences in the subjects' reactions to the program. 

The four-year-olds were most enthusiastic about the program, followed by the 
three- and then the f ivb-year-old subjects. The five-year-old males were 
least enthusiastic about the program and at all age levels girls responded 
overtly to questions, suggestions, and directions of the television "teacher" 
a greater proportion of the time than did boys. 

Based on overt behavior of viewers, the television program was best 
liked by the four-year-old subjects and was of more interest to girls than 
to boys. 

CONCLUSIONS AND IMPLICATIONS 

The use of systematic observational systems to provide formative 
evaluation of preschool television programs can be of significant value in 
guiding changes in presentation techniques, content, and emphasis. Given 
a larger number of observers, which in this study were paraprofessionals, 
or more specific needs, a much more sophisticated observational system could 
be developed to provide objective data on which to base these decisions. 

The television lessons broadcast during the second year (1969-70) of 
the ECE program held the children's interest a greater proportion of time 
than did the lessons telecast during the first program year. 
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ATTACHMENT 9-1 



Television Response Code Sheet 
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